Camera-radar fusion using correspondences

ABSTRACT

A method of performing sensor fusion of data obtained from a camera (51) and a supplemental sensor (52), the method comprising performing patch tracking (55) on image data (53) provided by the camera (51) to determine tracked patches (56), and performing a fusion (57) of the image data (53) obtained from the camera (51) with supplemental data (54) provided by the supplemental sensor (52) based on the tracked patches (56).

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to European Application No. 22166015.2,filed Mar. 31, 2022, the entire contents of which are incorporatedherein by reference.

TECHNICAL FIELD

The present disclosure generally pertains to the field of processingsensor data.

TECHNICAL BACKGROUND

In automotive, driver assistance systems (ADAS) are emerging for variousapplications. One major such application is obstacle detection forpreventing collisions.

Teck-Yian Lim et al in “Radar and Camera Early Fusion for VehicleDetection in Advanced Driver Assistance Systems”, 2019, describe aPerception module of a modern Advanced Driver Assistance Systems (ADAS).

Chang, et al in “Spatial Attention Fusion for Obstacle Detection UsingMmWave Radar and Vision Sensor”, Sensors 2020, 20, 956.https://doi.org/10.3390/s20040956 describe a spatial attention fusion(SAF) method for obstacle detection using mmWave radar and visionsensor.

Felix Nobis et al in “A Deep Learning-based Radar and Camera Sensor,Fusion Architecture for Object Detection”, arXiv:2005.07431, describe anenhancement of current 2D object detection networks by fusing cameradata and projected sparse radar data in the network layers.

Cars are commonly equipped with sensors such as cameras and radars.Different sensors, however, have different strengths and weaknesses.Monocular cameras, for instance, excel at measuring angular position,however they have no inherent capability of measuring distances. On theother hand, radar excels at measuring radial velocity and distance,quantities that cameras cannot measure. Radar, however, performs poorlyat measuring angular position. Often radar can estimate horizontalangles to some degree—albeit not as good as cameras—while the accuracyfor vertical angles is not good or it may even be not possible suchangles at all.

From that perspective, systems try to fuse information from bothsensors, trying to combine the strengths of either sensor. This conceptis known as sensor fusion.

It is generally desirable to provide better techniques for sensorfusion.

SUMMARY

According to a first aspect the disclosure provides a method ofperforming sensor fusion of data obtained from a camera and asupplemental sensor, the method comprising performing patch tracking onimage data provided by the camera to determine tracked patches, andperforming a fusion of the image data obtained from the camera withsupplemental data provided by the supplemental sensor based on thetracked patches.

According to a further aspect the disclosure provides a devicecomprising circuitry, the circuitry being configured to executeinstructions, the instructions, when executed on the circuitry,performing patch tracking on image data provided by a camera todetermine tracked patches, and a fusion of the image data obtained fromthe camera with supplemental data provided by a supplemental sensorbased on the tracked patches.

Further aspects are set forth in the dependent claims, the followingdescription and the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments are explained by way of example with respect to theaccompanying drawings, in which:

FIG. 1 provides a schematic representation of affine patch tracking andmatching;

FIG. 2 provides a schematic example of a set of patches obtained byaffine patch tracking performed on an image sequence of a scene;

FIG. 3 provides a schematic example of determining the scale ratio for apatch;

FIG. 4 shows an example of determining a Time to Collision from radardata;

FIG. 5 shows an example of a system for camera-radar fusion using affinecorrespondences;

FIG. 6 a schematically shows a process of camera-radar fusion usingaffine correspondences;

FIG. 6 b shows an example of how to identify, for each point of thepoint cloud, a nearby patch is using the patch scale ratio and theequivalent scale ratio;

FIG. 7 schematically shows a process of using information fromcamera-radar fusion on an object;

FIG. 8 a provides a table in which the performance of a camera iscompared to the performance of radar;

FIG. 8 b provides a table in which the performance of a camera thatexploits affine correspondences is compared to the performance of radar;

FIG. 9 is a block diagram depicting an example of schematicconfiguration of a vehicle control system; and

FIG. 10 is a diagram of assistance in explaining an example ofinstallation positions of an outside-vehicle information detectingsection and an imaging section.

DETAILED DESCRIPTION OF EMBODIMENTS

Before a detailed description of the embodiments under reference of FIG.1 , general explanations are made.

Camera and radar-based systems operate in very different domains:cameras on pixels in images, radar typically on sparse point clouds inthe world. The embodiments described below in more detail relate tosystems and processes for sensor fusion, and in particular to the fusionof information from camera and radar. The embodiments combine theadvantages of both sensor types.

The embodiments described below provide a natural domain in which radarand camera data can be fused. That is, despite that cameras and radaroperate in different domains, the embodiments allow a conversion betweenthe domains and they avoid ambiguities which can deteriorate the fusionresult. In particular, by applying the technology of the embodiments,inherent ambiguities of sensor fusion are resolved, such as not knowingthe distance when transforming image data to the world, and ambiguitiesin the location in the image when transforming radar data to the imageare avoided.

The embodiments described below in more detail disclose a method ofperforming sensor fusion of data obtained from a camera and asupplemental sensor, the method comprising performing patch tracking onimage data provided by the camera to determine tracked patches, andperforming a fusion of the image data obtained from the camera withsupplemental data provided by the supplemental sensor based on thetracked patches.

Sensor fusion may comprise any process of combining sensor data or dataderived from separate sources such that the resulting information hasless uncertainty than would be possible when these sources were usedindividually.

Performing patch tracking may comprises performing an affine patchtracking.

Affine patch tracking is based on the principle of affine transformationwhich in general describes an automorphism of an affine space (Euclideanspaces are specific affine spaces). An affine transformation may forexample be a function which maps an affine space onto itself whilepreserving both the dimension of any affine subspaces (meaning that itsends points to points, lines to lines, planes to planes, and so on) andthe ratios of the lengths of parallel line segments.

The method may comprise determining, for a patch of the tracked patches,a scale ratio. The scale ration may for example be a patch scale ration.

The method may comprise determining, for a part of the supplemental dataprovided by the supplemental sensor, an equivalent scale ratio.

The method may comprise identifying, for a part of the supplemental dataprovided by the supplemental sensor, a corresponding patch using a scaleratio of the patch and an equivalent scale ratio related to thesupplemental data provided by the supplemental sensor.

The method may comprise applying part of the supplemental data providedby the supplemental sensor to a patch of the tracked patches.

The supplemental data may for example be provided by the supplementalsensor in the form of a point cloud.

The part of the supplemental data may be a point of a point cloud.

According to some embodiments, the supplemental sensor is a radardevice. According to other embodiments, the supplemental sensor may be aLidar device or a Time of Flight device.

The equivalent scale ratio may for example be determined from distanceinformation and radial velocity information of a point of a radar pointcloud.

The method may further comprise, for a point that is considered beinglocated close to a patch of the tracked patches, comparing an equivalentscale ratio of the point with a scale ratio of the patch obtained fromaffine patch tracking to determine, if the scale ratios match.

The method may further comprise discarding those points of points thatare considered being located close to a patch of the tracked patches,whose equivalent scale ratios do not match with a scale ratio of thepatch obtained from affine patch tracking.

The method may further comprise performing object segmentation on theimage data captured by the camera.

The method may further comprise averaging supplemental data related topatches associated with an object, and associating the averagedinformation with the object.

The supplemental data may comprise Time to Collision information.

The sensor fusion is applied in an automotive context.

The embodiments also disclose a device comprising circuitry, thecircuitry being configured to execute instructions, the instructions,when executed on the circuitry, performing patch tracking on image dataprovided by a camera to determine tracked patches, and a fusion of theimage data obtained from the camera with supplemental data provided by asupplemental sensor based on the tracked patches.

Circuitry may include a processor, a memory (RAM, ROM or the like), astorage, input means (mouse, keyboard, camera, etc.), output means(display (e.g. liquid crystal, (organic) light emitting diode, etc.),loudspeakers, etc., a (wireless) interface, etc., as it is generallyknown for electronic devices (computers, smartphones, etc.). Moreover,circuitry may include sensors for sensing still image or video imagedata (image sensor, camera sensor, video sensor, etc.), etc.

The embodiments also disclose a system comprising a camera, asupplemental sensor, and the device of claim 16, the device beingconfigured to perform patch tracking on image data provided by thecamera to determine tracked patches, and to perform a fusion of theimage data obtained from the camera with supplemental data provided bythe supplemental sensor based on the tracked patches.

The embodiments also disclose a computer-implemented method comprisingperforming sensor fusion of data obtained from a camera and asupplemental sensor, the method comprising performing patch tracking onimage data provided by the camera to determine tracked patches, andperforming a fusion of the image data obtained from the camera withsupplemental data provided by the supplemental sensor based on thetracked patches. The computer-implemented method may compriseinstructions, which, when executed on a processor, perform theprocessing steps described in the embodiments.

The embodiments also disclose a machine-readable recording mediumcomprising instructions, which, when executed on a processor, performthe processing steps described in the embodiments.

Affine Patch Tracking

The embodiments described below in more detail make use of theadditional information provided by affine correspondences to provide agood domain for fusion camera and radar data. This allows for anefficient and robust way of fusing camera and radar measurements, whichcan be easily integrated in existing systems based on tracking affinecorrespondences.

In the case of affine correspondences, these features are not merelypixels but patches in the image. This gives major advantages overtracking pixels in terms of robustness. Affine correspondences arefrequently used in Simultaneous Localization and Mapping (SLAM) systems,but also in other object tracking systems.

Affine patch tracking has essentially two aspects: The first aspectconsists in the identification of an appropriate set of 2D (spatial)patches in image data to represent each surface in a scene.

The second aspect consists in the tracking of the patches through theimage sequence (see 53 in FIG. 5 ) provided by a camera (see 51 in FIG.5 ).

These techniques are well known to the skilled person. At a first step,points (or features or patches) are located in the images. Thesecorrespond to points where there is structure. By solving the flow(optical, affine, . . . ) equations, the flow on these points isestimated.

An overview of suitable techniques for object tracking and its affineextension is given by Khaled Jedoui in “Lecture 18: Tracking”(http://vision.stanford.edu/teaching/cs131_fall1718/files/18_notes.pdf).

Identifying the 2D patches may for example be based on identifyingdistinct surfaces in an image. Each 2D patch may for example relate to arectangular region in a first frame of an image sequence.

The tracking of the 2D patches and estimation of their associated affinemotion parameters may for example be achieved using weighted linearregression over an estimated optical flow field.

In addition to affine parametrization, a complete projectiveparametrization may be applied, which results in a complete homography.

For example, the technique described by Molton et al in “Locally PlanarPatch Features for Real-Time Structure from Motion”, 2004, In Proc.British Machine Vision Conf,(http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.422.1885) maybe applied. FIG. 1 provides a schematic representation of affine patchtracking and matching. A rectangular 2D patch 11 is identified in afirst frame (T=0) of an image sequence. By tracking, a corresponding 2Dpatch 12 is identified in a second frame (T=1) of the image sequence.image sequence. Still further, by tracking, a corresponding 2D patch 13is identified in a third frame (T=2) of the image sequence. Due tomotion of the camera or objects, 2D patches 12 and 13 that correspond topatch 11 are no longer rectangular, but have a shape that relates to theshape of patch 11 by affine transformation. By affine transformationsgeometric transformation such as lines and parallelism are preserved,but not necessarily distances and angles. By tracking 2D patches in theimage sequence over time, 2D patches identified at distinct times in theimage sequence are matched to each other.

Using technology such as affine SLAM, rectangular patches that areidentified and tracked in the 2D image sequence can be tracked in 3D.That is, each 2D patch can be related to a 3D patch and 3D informationsuch as 3D position of the points defining the patch and a surfacenormal may be attributed to each patch.

As described above, affine patch tracking gives additional informationon the features identified in an image sequence provided by a camera. Itbecomes possible to estimate a surface normal for the feature in worldcoordinates.

FIG. 2 provides a schematic example of a set of patches obtained byaffine patch tracking performed on an image sequence of a scene.

Patch Scale Ratio (Camera)

As affine patch tracking gives additional information on the featuresidentified in an image sequence provided by a camera, it becomespossible to estimate a size change of the patch between two frames(called here, the “scale ratio”).

This scale ratio does not give a direct indication of the scale, i.e.,the distance of the feature from the camera, but it provides indicationon how quickly a feature in the image sequence is approaching the camera(time to contact) or moving away from the camera (e.g., the time ittakes until the distance doubles), relative to its (unknown) distance.

This scale ratio is a quantity that is used for the fusion of sensordata in the sensor fusion described in the embodiments below in moredetail.

The scale ratio may be determined by comparing the dimensions of two 3Dpatched that have been matched by affine patch tracking. For example, ascale ratio may be determined from the distance of two matching pointsof a tracked patch over time in camera coordinates.

FIG. 3 provides a schematic example of determining the scale ratio for apatch. Affine patch tracking provides a 3D patch 31 at time T1, and amatching 3D patch 32 at time T2. Affine patch tracking provides matchingpoints P₁ and P₂ of patches 31 and 32. A distance d1 between points P₁and P₂ at time T1 is obtained from their positions P₁ (T1) and,respectively, P₂ (T1) at time T1 in world coordinates. A distance d2between points P₁ and P₂ at time T2 is obtained from their positionsP₁(T2) and, respectively, P₂(T2) at time T2 in world coordinates. Ascale ratio can be determined from the affine patch tracking byrelating, for example, the distance d1 at time T1 to the distance d2 attime T2, according to:

$\begin{matrix}{{SR}_{affine} = \frac{d2}{d1}} & {{Eq}.(1)}\end{matrix}$

In the embodiment described above, a scale ratio results from thedistance of two matching points of a tracked patch over time. Theembodiments are, however, not limited to determining the scale ratiofrom two points. Any geometric feature that represents the scale of apatch can be used to determine the scale ratio, for example distancesbetween multiple points of a patch, a surface of the patch, or the like.

The determination of the scale ratio according to the example aboveassumes that there is no rotation or that there is the same scale on allthe sides of the patches. A more general description is the following:

An affine patch change can be represent by an homography matrix

$\begin{matrix}{H = \begin{bmatrix}h_{11} & h_{12} & h_{13} \\h_{21} & h_{22} & h_{23} \\h_{31} & h_{32} & h_{33}\end{bmatrix}} & {{Eq}.(2)}\end{matrix}$

An area ratio AR, i.e. the change in the patch area in the image, isgiven by the determinant of the upper left sub matrix

$\begin{matrix}{{AR} = {{❘\begin{bmatrix}h_{11} & h_{12} \\h_{21} & h_{22}\end{bmatrix}❘} = {{h_{11} \cdot h_{22}} - {h_{12} \cdot h_{21}}}}} & {{Eq}.(3)}\end{matrix}$

A scale ratio SR_(affine) may be obtained directly from this area ratioAR:

SR _(affine) =AR  Eq. (4).

Alternatively, a scale ratio may be obtained by taking the square rootof the area ratio

SR _(affine) =√{square root over (AR)}.  Eq. (5).

Equivalent Scale Ratio (Radar)

Assuming an exemplifying case in which a radar device provides its datain the form of a point cloud, where each point of the point cloud has adistance z (relative to the camera plane), and a radial velocity v,where positive is away from a camera used for (extrinsic) calibration(If the relative position and orientation of the radar and camera toeach other are known, the position of a point in the radar point cloudcan be determined relative to the camera, i.e. in “camera coordinates”)

Assuming that z1 and z2 are the distances (relative to the camera plane)of a point p in the world in two subsequent frames, and point p hasradial velocity v, and dt is the time between the frames, then:

z1=z2−v·dt  Eq. (6)

The scale of a feature in an image of the scene is inverselyproportional to its distance z, therefore a scale ratio SR_(equiv) fromframe 1 to frame 2 can be obtained from radar measurements z1, z2 and vaccording to:

$\begin{matrix}{{SR}_{equiv} = {\frac{\frac{1}{z2}}{\frac{1}{z1}} = {\frac{z1}{z2} = {\frac{{z2} - {v \cdot {dt}}}{z2} = {1 - {{dt} \cdot \frac{v}{z2}}}}}}} & {{Eq}.(7)}\end{matrix}$

Since scale ratios are computed from previous frames the derivation isdone in terms of z2, not z1.

By matching the scale ratios SR_(affine) of patches obtained by affinepatch tracking with equivalent scale ratios SR_(equiv) of the pointsobtained from radar data, information obtained from radar can beassociated with corresponding patches identified by affine patchtracking. In this way, for example radial velocity information, or anyother information derived from radial velocity information may beattributed to patches.

FIG. 4 shows an example of determining a Time to Collision from radardata. At time t, a vehicle 42 is located a distance d away from a camera41. The vehicle 42 approaches the camera 41 with radial velocity v. ATime to Collision TTC is obtained from distance d and velocity vaccording to:

TTC=d/v  Eq. (8)

The Time to Collision is related to the scale ratio by:

TTC ⁻¹ =v/x(t)=∂ LOG(x(t))∂t  Eq. (9)

TTC ⁻¹=(SR−1)·fps  Eq. (10)

where fps is the number of frames per second of an image sequenceprovided by a camera.

Camera-Radar Fusion Using Affine Correspondences

Using affine correspondences, the embodiments described here provide asuitable domain for fusing data from both sensor types, camera andradar. In particular, the embodiments provide a well-defined commondomain for camera and radar systems in which camera and radar data canbe fused.

FIG. 5 shows an example of a system for camera-radar fusion using affinecorrespondences. A camera 51 produces image data 53. This camera 51 mayfor example be part of a imaging section of a vehicle (7410 in FIG. 9 ).This image data 53 provided by camera 51 typically comprises a sequenceof images obtained from a scene. The image data 53 is processed byaffine patch tracking 55 to identify and track patches in the image data53. Based on this process, affine patch tracking 55 generates trackedpatches 56. A radar device 52 produces radar data 54. This radar device52 may for example be part of an outside-vehicle information detectingsection (7420 in FIG. 9 ). A camera-radar fusion 57 receives the imagedata 53 from the camera 51, the tracked patches 56 from the affine patchtracking 55, and the radar data 54 from the radar 52, and performssensor fusion of the image data 53 and the radar data 54 based oninformation from the tracked patches 56. This camera-radar fusion 57 isdescribed in more detail below with regard to FIG. 6 .

In the embodiment of FIG. 5 , a radar device is shown as a supplementalsensor 52 that provides supplemental data 54 for sensor fusion 57. Thesupplemental data 54 provided by the supplemental sensor 52 isaccordingly denoted as radar data, and the sensor fusion 57 isaccordingly denoted as camera-radar fusion.

The supplemental data may for example take the form of a point cloud. Inthe embodiments described below in more detail, a radar point cloud isdescribed as an example of supplemental data. The supplemental data may,however, take other form than a point cloud (for example usingpre-aggregation of the data, or the like).

FIG. 6 a schematically shows a process of camera-radar fusion usingaffine correspondences. The process may for example be operated by acontrol unit (7600 in FIG. 9 ) of a vehicle control system (7000 in FIG.9 ). At 61, it is determined, for each patch of the tracked patches (56in FIG. 5 ), a patch scale ratio from the distance of two matchingpoints of a respective patch. This process is described in more detailabove with regard to FIG. 3 and Eq. (1). At 62, it is determined, foreach point of a radar point cloud, an equivalent scale ratio from thedistance information and radial velocity information of a respectivepoint of the radar point cloud. This process is described in more detailabove with regard to Eq. (3). At 63, for each point of the point cloud,a nearby patch is identified using the patch scale ratio and theequivalent scale ratio. The point and the nearby patch witch matchingscale ratio are assigned to each other. At 64, for each patch, theassociated radar information (e.g. distance and velocity) of the mappedpoints is applied to the patch.

FIG. 6 b shows an example of how to identify, for each point of thepoint cloud, a nearby patch is using the patch scale ratio and theequivalent scale ratio. At 65, the distance between a patch and pointsof the point cloud is determined. At 66, based on the distances obtainedat 65, those points of the point cloud are identified that are close tothe patch. At 67, for each point of the points that are considered asbeing located close to the patch, a comparison of the equivalent scaleratio of the point with the scale ratio of the patch obtained fromaffine patch tracking is performed to determine, if the scale ratiosmatch. At 68, those points, whose equivalent scale ratios do not matchwith the scale ratio of the patch obtained from affine patch tracking,are discarded.

Determining, at 65, the distance between a patch and a point of thepoint cloud may for example be achieved by determining a patch positionby averaging the world coordinates of all points defining the patch anddetermining the distance between the patch position and the worldcoordinate of the point of the point cloud.

Identifying, at 66, based on the distances obtained at 65, those pointsof the point cloud that are close to the patch may for example achievedby comparing the distances obtained at 65 with a predefined thresholdvalue and considering those points as close to the patch whose distanceto the patch is below the threshold value.

Performing, at 67, for each point of the points that are considered asbeing located close to the patch, a comparison of the equivalent scaleratio of the point with the scale ratio of the patch obtained fromaffine patch tracking may for example be achieved by subtracting thescale ratios to determine a scale ratio difference, and comparing theabsolute value of the scale ratio difference with a predefined thresholdvalue.

With the concept as described in the embodiments, it is not necessary toconvert data from one sensor to the domain of the other and findcorrespondences between the data.

Besides a robust way of tracking features in an image sequence, theyprovide additional information on the features such as their scaleratio. For example, tracking sparse features over time in an imagesequence is a frequently used technique in computer vision in general.

Radar does give a measurement on the absolute distance, and on theradial velocity. From these two quantities we can compute an impliedscale ratio, since knowing radial distance and velocity, we can estimatee.g. the time to contact and from this the scale ratio. This gives us acommon domain for radar and camera, which we can use for relatingtracked affine correspondences to radar measurements, eliminating muchof the ambiguities arising in the conversion between camera and radardomains.

The main advantage of the invention is that it resolves much of thedomain ambiguity between camera and radar sensors. By that, it allowsfor more straightforward and precise fusion.

Furthermore, it can be integrated easily in existing vision-basedsystems based on affine correspondences. This integration comes atnegligible extra computation time or memory requirements and istherefore well-suited for real-time capable systems.

Object Segmentation

FIG. 7 schematically shows a process of using information fromcamera-radar fusion on an object level. The process may for example beoperated by a control unit (7600 in FIG. 9 ) of a vehicle control system(7000 in FIG. 9 ). At 71, affine patch tracking is performed on an imagesequence obtained by a camera. At 72, it is performed an objectsegmentation of scene captured by camera. At 73, the patches obtained byaffine patch tracking 71 are associated with objects obtained by objectsegmentation 72. At 74, for each object identified by objectsegmentation 72, the supplemental information of all patches associatedwith the object is average, and the averaged information is associatedwith the object.

Performing object segmentation of scene captured by camera, at 72, mayfor example be achieved by any techniques known to the skilled person,for example the technique presented by Di Feng et al in “DeepMulti-modal Object Detection and Semantic Segmentation for AutonomousDriving: Datasets, Methods, and Challenges”, arXiv:1902.07830.

Associating, at 73, the patches obtained by affine patch tracking withobjects obtained by object segmentation may for example be achieved byfinding, for each patch, a corresponding object that is close to thepatch.

Averaging, at 74 the supplemental information of all patches associatedwith the object may for example comprise averaging distance and radialspeed information obtained from radar over all patches that are relatedto an object. Also, other information obtained from the radar domain,such as Time to Collision (TTC) may be averaged over patches andattributed to an object at the object level.

Performance of Camera-Radar Fusion

FIG. 8 a provides a table in which the performance of a camera iscompared to the performance of radar. For angular position, theperformance of a camera is very good, the performance of radar is,however, not so good. For radial distance, the performance of a camerais not so good, the performance of radar is, however, good. For radialspeed, a camera provides no information, the performance of radar is,however, very good.

FIG. 8 b provides a table in which the performance of a camera thatexploits affine correspondences is compared to the performance of radar.For angular position, the performance of a camera with affinecorrespondences is very good, the performance of radar is, however, notso good. For radial distance, the performance of a camera with affinecorrespondences is not so good, the performance of radar is, however,good. For radial speed, a camera with affine correspondences provides noinformation, the performance of radar is, however, very good. A camerawith affine correspondences provides a good performance in determining ascale ratio (SR) and a Time to Contact (TTC). This information is alsoimplicitly contained in radar data. A camera with affine correspondencesprovides a good performance in determining a surface normal. However,radar does not provide any information on surface normals.

Implementation

The technology according to an embodiment of the present disclosure isapplicable to various products. The techniques of the embodiments mayfor example be used for driving assistance systems. For example, thetechnology according to an embodiment of the present disclosure may beimplemented as a device included in a mobile body that is any of kindsof automobiles, electric vehicles, hybrid electric vehicles,motorcycles, bicycles, personal mobility vehicles, airplanes, drones,ships, robots, construction machinery, agricultural machinery(tractors), and the like.

FIG. 9 is a block diagram depicting an example of schematicconfiguration of a vehicle control system 7000 as an example of a mobilebody control system to which the technology according to an embodimentsof the present disclosure can be applied. The vehicle control system7000 includes a plurality of electronic control units connected to eachother via a communication network 7010. In the example depicted in FIG.9 , the vehicle control system 7000 includes a driving system controlunit 7100, a body system control unit 7200, a battery control unit 7300,an outside-vehicle information detecting unit 7400, an in-vehicleinformation detecting unit 7500, and an integrated control unit 7600.The communication network 7010 connecting the plurality of control unitsto each other may, for example, be a vehicle-mounted communicationnetwork compliant with an arbitrary standard such as controller areanetwork (CAN), local interconnect network (LIN), local area network(LAN), FlexRay (registered trademark), or the like.

Each of the control units includes: a microcomputer that performsarithmetic processing according to various kinds of programs; a storagesection that stores the programs executed by the microcomputer,parameters used for various kinds of operations, or the like; and adriving circuit that drives various kinds of control target devices.Each of the control units further includes: a network interface (I/F)for performing communication with other control units via thecommunication network 7010; and a communication I/F for performingcommunication with a device, a sensor, or the like within and withoutthe vehicle by wire communication or radio communication. A functionalconfiguration of the integrated control unit 7600 illustrated in FIG. 9includes a microcomputer 7610, a general-purpose communication I/F 7620,a dedicated communication I/F 7630, a positioning section 7640, a beaconreceiving section 7650, an in-vehicle device I/F 7660, a sound/imageoutput section 7670, a vehicle-mounted network I/F 7680, and a storagesection 7690. The other control units similarly include a microcomputer,a communication I/F, a storage section, and the like.

The driving system control unit 7100 controls the operation of devicesrelated to the driving system of the vehicle in accordance with variouskinds of programs. For example, the driving system control unit 7100functions as a control device for a driving force generating device forgenerating the driving force of the vehicle, such as an internalcombustion engine, a driving motor, or the like, a driving forcetransmitting mechanism for transmitting the driving force to wheels, asteering mechanism for adjusting the steering angle of the vehicle, abraking device for generating the braking force of the vehicle, and thelike. The driving system control unit 7100 may have a function as acontrol device of an antilock brake system (ABS), electronic stabilitycontrol (ESC), or the like.

The driving system control unit 7100 is connected with a vehicle statedetecting section 7110. The vehicle state detecting section 7110, forexample, includes at least one of a gyro sensor that detects the angularvelocity of axial rotational movement of a vehicle body, an accelerationsensor that detects the acceleration of the vehicle, and sensors fordetecting an amount of operation of an accelerator pedal, an amount ofoperation of a brake pedal, the steering angle of a steering wheel, anengine speed or the rotational speed of wheels, and the like. Thedriving system control unit 7100 performs arithmetic processing using asignal input from the vehicle state detecting section 7110, and controlsthe internal combustion engine, the driving motor, an electric powersteering device, the brake device, and the like.

The body system control unit 7200 controls the operation of variouskinds of devices provided to the vehicle body in accordance with variouskinds of programs. For example, the body system control unit 7200functions as a control device for a keyless entry system, a smart keysystem, a power window device, or various kinds of lamps such as aheadlamp, a backup lamp, a brake lamp, a turn signal, a fog lamp, or thelike. In this case, radio waves transmitted from a mobile device as analternative to a key or signals of various kinds of switches can beinput to the body system control unit 7200. The body system control unit7200 receives these input radio waves or signals, and controls a doorlock device, the power window device, the lamps, or the like of thevehicle.

The battery control unit 7300 controls a secondary battery 7310, whichis a power supply source for the driving motor, in accordance withvarious kinds of programs. For example, the battery control unit 7300 issupplied with information about a battery temperature, a battery outputvoltage, an amount of charge remaining in the battery, or the like froma battery device including the secondary battery 7310. The batterycontrol unit 7300 performs arithmetic processing using these signals,and performs control for regulating the temperature of the secondarybattery 7310 or controls a cooling device provided to the battery deviceor the like.

The outside-vehicle information detecting unit 7400 detects informationabout the outside of the vehicle including the vehicle control system7000. For example, the outside-vehicle information detecting unit 7400is connected with at least one of an imaging section 7410 and anoutside-vehicle information detecting section 7420. The imaging section7410 includes at least one of a time-of-flight (ToF) camera, a stereocamera, a monocular camera, an infrared camera, and other cameras. Theoutside-vehicle information detecting section 7420, for example,includes at least one of an environmental sensor for detecting currentatmospheric conditions or weather conditions and a peripheralinformation detecting sensor for detecting another vehicle, an obstacle,a pedestrian, or the like on the periphery of the vehicle including thevehicle control system 7000.

The environmental sensor, for example, may be at least one of a raindrop sensor detecting rain, a fog sensor detecting a fog, a sunshinesensor detecting a degree of sunshine, and a snow sensor detecting asnowfall. The peripheral information detecting sensor may be at leastone of an ultrasonic sensor, a radar device, and a LIDAR device (Lightdetection and Ranging device, or Laser imaging detection and rangingdevice). Each of the imaging section 7410 and the outside-vehicleinformation detecting section 7420 may be provided as an independentsensor or device, or may be provided as a device in which a plurality ofsensors or devices are integrated.

FIG. 10 depicts an example of installation positions of the imagingsection 7410 and the outside-vehicle information detecting section 7420.Imaging sections 7910, 7912, 7914, 7916, and 7918 are, for example,disposed at at least one of positions on a front nose, sideview mirrors,a rear bumper, and a back door of the vehicle 7900 and a position on anupper portion of a windshield within the interior of the vehicle. Theimaging section 7910 provided to the front nose and the imaging section7918 provided to the upper portion of the windshield within the interiorof the vehicle obtain mainly an image of the front of the vehicle 7900.The imaging sections 7912 and 7914 provided to the sideview mirrorsobtain mainly an image of the sides of the vehicle 7900. The imagingsection 7916 provided to the rear bumper or the back door obtains mainlyan image of the rear of the vehicle 7900. The imaging section 7918provided to the upper portion of the windshield within the interior ofthe vehicle is used mainly to detect a preceding vehicle, a pedestrian,an obstacle, a signal, a traffic sign, a lane, or the like.

Incidentally, FIG. 10 depicts an example of photographing ranges of therespective imaging sections 7910, 7912, 7914, and 7916. An imaging rangea represents the imaging range of the imaging section 7910 provided tothe front nose. Imaging ranges b and c respectively represent theimaging ranges of the imaging sections 7912 and 7914 provided to thesideview mirrors. An imaging range d represents the imaging range of theimaging section 7916 provided to the rear bumper or the back door. Abird's-eye image of the vehicle 7900 as viewed from above can beobtained by superimposing image data imaged by the imaging sections7910, 7912, 7914, and 7916, for example.

Outside-vehicle information detecting sections 7920, 7922, 7924, 7926,7928, and 7930 provided to the front, rear, sides, and corners of thevehicle 7900 and the upper portion of the windshield within the interiorof the vehicle may be, for example, an ultrasonic sensor or a radardevice. The outside-vehicle information detecting sections 7920, 7926,and 7930 provided to the front nose of the vehicle 7900, the rearbumper, the back door of the vehicle 7900, and the upper portion of thewindshield within the interior of the vehicle may be a LIDAR device, forexample. These outside-vehicle information detecting sections 7920 to7930 are used mainly to detect a preceding vehicle, a pedestrian, anobstacle, or the like.

Returning to FIG. 9 , the description will be continued. Theoutside-vehicle information detecting unit 7400 makes the imagingsection 7410 image an image of the outside of the vehicle, and receivesimaged image data. In addition, the outside-vehicle informationdetecting unit 7400 receives detection information from theoutside-vehicle information detecting section 7420 connected to theoutside-vehicle information detecting unit 7400. In a case where theoutside-vehicle information detecting section 7420 is an ultrasonicsensor, a radar device, or a LIDAR device, the outside-vehicleinformation detecting unit 7400 transmits an ultrasonic wave, anelectromagnetic wave, or the like, and receives information of areceived reflected wave. On the basis of the received information, theoutside-vehicle information detecting unit 7400 may perform processingof detecting an object such as a human, a vehicle, an obstacle, a sign,a character on a road surface, or the like, or processing of detecting adistance thereto. The outside-vehicle information detecting unit 7400may perform environment recognition processing of recognizing arainfall, a fog, road surface conditions, or the like on the basis ofthe received information. The outside-vehicle information detecting unit7400 may calculate a distance to an object outside the vehicle on thebasis of the received information.

In addition, on the basis of the received image data, theoutside-vehicle information detecting unit 7400 may perform imagerecognition processing of recognizing a human, a vehicle, an obstacle, asign, a character on a road surface, or the like, or processing ofdetecting a distance thereto. The outside-vehicle information detectingunit 7400 may subject the received image data to processing such asdistortion correction, alignment, or the like, and combine the imagedata imaged by a plurality of different imaging sections 7410 togenerate a bird's-eye image or a panoramic image. The outside-vehicleinformation detecting unit 7400 may perform viewpoint conversionprocessing using the image data imaged by the imaging section 7410including the different imaging parts.

The in-vehicle information detecting unit 7500 detects information aboutthe inside of the vehicle. The in-vehicle information detecting unit7500 is, for example, connected with a driver state detecting section7510 that detects the state of a driver. The driver state detectingsection 7510 may include a camera that images the driver, a biosensorthat detects biological information of the driver, a microphone thatcollects sound within the interior of the vehicle, or the like. Thebiosensor is, for example, disposed in a seat surface, the steeringwheel, or the like, and detects biological information of an occupantsitting in a seat or the driver holding the steering wheel. On the basisof detection information input from the driver state detecting section7510, the in-vehicle information detecting unit 7500 may calculate adegree of fatigue of the driver or a degree of concentration of thedriver, or may determine whether the driver is dozing. The in-vehicleinformation detecting unit 7500 may subject an audio signal obtained bythe collection of the sound to processing such as noise cancelingprocessing or the like.

The integrated control unit 7600 controls general operation within thevehicle control system 7000 in accordance with various kinds ofprograms. The integrated control unit 7600 is connected with an inputsection 7800. The input section 7800 is implemented by a device capableof input operation by an occupant, such, for example, as a touch panel,a button, a microphone, a switch, a lever, or the like. The integratedcontrol unit 7600 may be supplied with data obtained by voicerecognition of voice input through the microphone. The input section7800 may, for example, be a remote control device using infrared rays orother radio waves, or an external connecting device such as a mobiletelephone, a personal digital assistant (PDA), or the like that supportsoperation of the vehicle control system 7000. The input section 7800 maybe, for example, a camera. In that case, an occupant can inputinformation by gesture. Alternatively, data may be input which isobtained by detecting the movement of a wearable device that an occupantwears. Further, the input section 7800 may, for example, include aninput control circuit or the like that generates an input signal on thebasis of information input by an occupant or the like using theabove-described input section 7800, and which outputs the generatedinput signal to the integrated control unit 7600. An occupant or thelike inputs various kinds of data or gives an instruction for processingoperation to the vehicle control system 7000 by operating the inputsection 7800.

The storage section 7690 may include a read only memory (ROM) thatstores various kinds of programs executed by the microcomputer and arandom access memory (RAM) that stores various kinds of parameters,operation results, sensor values, or the like. In addition, the storagesection 7690 may be implemented by a magnetic storage device such as ahard disc drive (HDD) or the like, a semiconductor storage device, anoptical storage device, a magneto-optical storage device, or the like.

The general-purpose communication I/F 7620 is a communication I/F usedwidely, which communication I/F mediates communication with variousapparatuses present in an external environment 7750. The general-purposecommunication I/F 7620 may implement a cellular communication protocolsuch as global system for mobile communications (GSM (registeredtrademark)), worldwide interoperability for microwave access (WiMAX(registered trademark)), long term evolution (LTE (registeredtrademark)), LTE-advanced (LTE-A), or the like, or another wirelesscommunication protocol such as wireless LAN (referred to also aswireless fidelity (Wi-Fi (registered trademark)), Bluetooth (registeredtrademark), or the like. The general-purpose communication I/F 7620 may,for example, connect to an apparatus (for example, an application serveror a control server) present on an external network (for example, theInternet, a cloud network, or a company-specific network) via a basestation or an access point. In addition, the general-purposecommunication I/F 7620 may connect to a terminal present in the vicinityof the vehicle (which terminal is, for example, a terminal of thedriver, a pedestrian, or a store, or a machine type communication (MTC)terminal) using a peer to peer (P2P) technology, for example.

The dedicated communication I/F 7630 is a communication I/F thatsupports a communication protocol developed for use in vehicles. Thededicated communication I/F 7630 may implement a standard protocol such,for example, as wireless access in vehicle environment (WAVE), which isa combination of institute of electrical and electronic engineers (IEEE)802.11p as a lower layer and IEEE 1609 as a higher layer, dedicatedshort range communications (DSRC), or a cellular communication protocol.The dedicated communication I/F 7630 typically carries out V2Xcommunication as a concept including one or more of communicationbetween a vehicle and a vehicle (Vehicle to Vehicle), communicationbetween a road and a vehicle (Vehicle to Infrastructure), communicationbetween a vehicle and a home (Vehicle to Home), and communicationbetween a pedestrian and a vehicle (Vehicle to Pedestrian).

The positioning section 7640, for example, performs positioning byreceiving a global navigation satellite system (GNSS) signal from a GNSSsatellite (for example, a GPS signal from a global positioning system(GPS) satellite), and generates positional information including thelatitude, longitude, and altitude of the vehicle. Incidentally, thepositioning section 7640 may identify a current position by exchangingsignals with a wireless access point, or may obtain the positionalinformation from a terminal such as a mobile telephone, a personalhandyphone system (PHS), or a smart phone that has a positioningfunction.

The beacon receiving section 7650, for example, receives a radio wave oran electromagnetic wave transmitted from a radio station installed on aroad or the like, and thereby obtains information about the currentposition, congestion, a closed road, a necessary time, or the like.Incidentally, the function of the beacon receiving section 7650 may beincluded in the dedicated communication I/F 7630 described above.

The in-vehicle device I/F 7660 is a communication interface thatmediates connection between the microcomputer 7610 and variousin-vehicle devices 7760 present within the vehicle. The in-vehicledevice I/F 7660 may establish wireless connection using a wirelesscommunication protocol such as wireless LAN, Bluetooth (registeredtrademark), near field communication (NFC), or wireless universal serialbus (WUSB). In addition, the in-vehicle device I/F 7660 may establishwired connection by universal serial bus (USB), high-definitionmultimedia interface (HDMI (registered trademark)), mobilehigh-definition link (MHL), or the like via a connection terminal (and acable if necessary) not depicted in the figures. The in-vehicle devices7760 may, for example, include at least one of a mobile device and awearable device possessed by an occupant and an information devicecarried into or attached to the vehicle. The in-vehicle devices 7760 mayalso include a navigation device that searches for a path to anarbitrary destination. The in-vehicle device I/F 7660 exchanges controlsignals or data signals with these in-vehicle devices 7760.

The vehicle-mounted network I/F 7680 is an interface that mediatescommunication between the microcomputer 7610 and the communicationnetwork 7010. The vehicle-mounted network I/F 7680 transmits andreceives signals or the like in conformity with a predetermined protocolsupported by the communication network 7010.

The microcomputer 7610 of the integrated control unit 7600 controls thevehicle control system 7000 in accordance with various kinds of programson the basis of information obtained via at least one of thegeneral-purpose communication I/F 7620, the dedicated communication I/F7630, the positioning section 7640, the beacon receiving section 7650,the in-vehicle device I/F 7660, and the vehicle-mounted network I/F7680. For example, the microcomputer 7610 may calculate a control targetvalue for the driving force generating device, the steering mechanism,or the braking device on the basis of the obtained information about theinside and outside of the vehicle, and output a control command to thedriving system control unit 7100. For example, the microcomputer 7610may perform cooperative control intended to implement functions of anadvanced driver assistance system (ADAS) which functions includecollision avoidance or shock mitigation for the vehicle, followingdriving based on a following distance, vehicle speed maintainingdriving, a warning of collision of the vehicle, a warning of deviationof the vehicle from a lane, or the like. In addition, the microcomputer7610 may perform cooperative control intended for automatic driving,which makes the vehicle to travel autonomously without depending on theoperation of the driver, or the like, by controlling the driving forcegenerating device, the steering mechanism, the braking device, or thelike on the basis of the obtained information about the surroundings ofthe vehicle.

The microcomputer 7610 may generate three-dimensional distanceinformation between the vehicle and an object such as a surroundingstructure, a person, or the like, and generate local map informationincluding information about the surroundings of the current position ofthe vehicle, on the basis of information obtained via at least one ofthe general-purpose communication I/F 7620, the dedicated communicationI/F 7630, the positioning section 7640, the beacon receiving section7650, the in-vehicle device I/F 7660, and the vehicle-mounted networkI/F 7680. In addition, the microcomputer 7610 may predict danger such ascollision of the vehicle, approaching of a pedestrian or the like, anentry to a closed road, or the like on the basis of the obtainedinformation, and generate a warning signal. The warning signal may, forexample, be a signal for producing a warning sound or lighting a warninglamp.

The sound/image output section 7670 transmits an output signal of atleast one of a sound and an image to an output device capable ofvisually or auditorily notifying information to an occupant of thevehicle or the outside of the vehicle. In the example of FIG. 9 , anaudio speaker 7710, a display section 7720, and an instrument panel 7730are illustrated as the output device. The display section 7720 may, forexample, include at least one of an on-board display and a head-updisplay. The display section 7720 may have an augmented reality (AR)display function. The output device may be other than these devices, andmay be another device such as headphones, a wearable device such as aneyeglass type display worn by an occupant or the like, a projector, alamp, or the like. In a case where the output device is a displaydevice, the display device visually displays results obtained by variouskinds of processing performed by the microcomputer 7610 or informationreceived from another control unit in various forms such as text, animage, a table, a graph, or the like. In addition, in a case where theoutput device is an audio output device, the audio output deviceconverts an audio signal constituted of reproduced audio data or sounddata or the like into an analog signal, and auditorily outputs theanalog signal.

Incidentally, at least two control units connected to each other via thecommunication network 7010 in the example depicted in FIG. 9 may beintegrated into one control unit. Alternatively, each individual controlunit may include a plurality of control units. Further, the vehiclecontrol system 7000 may include another control unit not depicted in thefigures. In addition, part or the whole of the functions performed byone of the control units in the above description may be assigned toanother control unit. That is, predetermined arithmetic processing maybe performed by any of the control units as long as information istransmitted and received via the communication network 7010. Similarly,a sensor or a device connected to one of the control units may beconnected to another control unit, and a plurality of control units maymutually transmit and receive detection information via thecommunication network 7010.

Incidentally, a computer program for realizing the functions of affinepatch tracking (55 in FIG. 5 ) and sensor fusion (e.g. Camera-Radarfusion 57 in FIG. 5 ) can be implemented in one of the control units orthe like. In addition, a computer readable recording medium storing sucha computer program can also be provided. The recording medium is, forexample, a magnetic disk, an optical disk, a magneto-optical disk, aflash memory, or the like. In addition, the above-described computerprogram may be distributed via a network, for example, without therecording medium being used.

In the vehicle control system 7000 described above, the functions ofaffine patch tracking (55 in FIG. 5 ) and sensor fusion (e.g.Camera-Radar fusion 57 in FIG. 5 ) according to the embodimentsdescribed above can be applied to the integrated control unit 7600 inthe application example depicted in FIG. 9 .

It should be noted that the description above is only an exampleconfiguration. Alternative configurations may be implemented withadditional or other units, sensors, or the like.

It should also be noted that the division of the systems of FIGS. 5 and9 into units is only made for illustration purposes and that the presentdisclosure is not limited to any specific division of functions inspecific units.

It should also be recognized that the embodiments describe methods withan exemplary ordering of method steps. The specific ordering of methodsteps is, however, given for illustrative purposes only and should notbe construed as binding.

All units and entities described in this specification and claimed inthe appended claims can, if not stated otherwise, be implemented asintegrated circuit logic, for example, on a chip, and functionalityprovided by such units and entities can, if not stated otherwise, beimplemented by software.

In so far as the embodiments of the disclosure described above areimplemented, at least in part, using software-controlled data processingapparatus, it will be appreciated that a computer program providing suchsoftware control and a transmission, storage or other medium by whichsuch a computer program is provided are envisaged as aspects of thepresent disclosure.

Note that the present technology can also be configured as describedbelow:

-   -   (1) A method of performing sensor fusion of data obtained from a        camera and a supplemental sensor, the method comprising        performing patch tracking on image data provided by the camera        to determine tracked patches, and performing a fusion of the        image data obtained from the camera with supplemental data        provided by the supplemental sensor based on the tracked        patches.    -   (2) The method of (1), wherein performing patch tracking        comprises performing an affine patch tracking.    -   (3) The method of (1) or (2), comprising determining, for a        patch of the tracked patches, a patch scale ratio.    -   (4) The method of any one of (1) to (3), comprising determining,        for a part of the supplemental data provided by the supplemental        sensor, an equivalent scale ratio.    -   (5) The method of any one of (1) to (4), comprising identifying,        for a part of the supplemental data provided by the supplemental        sensor, a corresponding patch using a scale ratio of the patch        and an equivalent scale ratio related to the supplemental data        provided by the supplemental sensor.    -   (6) The method of any one of (1) to (5), comprising applying        part of the supplemental data provided by the supplemental        sensor to a patch of the tracked patches.    -   (7) The method of any one of (1) to (6), wherein the        supplemental data is provided by the supplemental sensor in the        form of a point cloud.    -   (8) The method of any one of (4) to (7), wherein the part of the        supplemental data is a point of a point cloud.    -   (9) The method of any one of (1) to (8), wherein the        supplemental sensor is a radar sensor.    -   (10) The method of any one of (4) to (9), wherein the equivalent        scale ratio is determined from distance information and radial        velocity information of a point of the radar point cloud.    -   (11) The method of any one of (1) to (10), comprising, for a        point that is considered being located close to a patch of the        tracked patches, comparing an equivalent scale ratio of the        point with a scale ratio of the patch obtained from patch        tracking to determine, if the scale ratios match.    -   (12) The method of any one of (1) to (11), comprising discarding        those points of points that are considered being located close        to a patch of the tracked patches, whose equivalent scale ratios        do not match with a scale ratio of the patch obtained from        affine patch tracking.    -   (13) The method of any one of (1) to (12), comprising performing        object segmentation on the image data captured by the camera.    -   (14) The method of any one of (1) to (13), comprising averaging        supplemental data related to patches associated with an object,        and associating the averaged information with the object.    -   (15) The method of any one of (1) to (14), wherein the        supplemental data comprises Time to Collision information.    -   (16) The method of any one of (1) to (15), wherein the sensor        fusion is applied in an automotive context.    -   (17) A device comprising circuitry, the circuitry being        configured to execute instructions, the instructions, when        executed on the circuitry, performing        -   patch tracking on image data provided by a camera to            determine tracked patches, and a fusion of the image data            obtained from the camera with supplemental data provided by            a supplemental sensor based on the tracked patches.    -   (18) A system comprising a camera, a supplemental sensor, and        the device of (17), the device being configured to perform patch        tracking on image data provided by the camera to determine        tracked patches, and to perform a fusion of the image data        obtained from the camera with supplemental data provided by the        supplemental sensor based on the tracked patches.

1. A method of performing sensor fusion of data obtained from a cameraand a supplemental sensor, the method comprising performing patchtracking on image data provided by the camera to determine trackedpatches, and performing a fusion of the image data obtained from thecamera with supplemental data provided by the supplemental sensor basedon the tracked patches.
 2. The method of claim 1, wherein performingpatch tracking comprises performing an affine patch tracking.
 3. Themethod of claim 1, comprising determining, for a patch of the trackedpatches, a scale ratio.
 4. The method of claim 1, comprisingdetermining, for a part of the supplemental data provided by thesupplemental sensor, an equivalent scale ratio.
 5. The method of claim1, comprising identifying, for a part of the supplemental data providedby the supplemental sensor, a corresponding patch using a scale ratio ofthe patch and an equivalent scale ratio related to the supplemental dataprovided by the supplemental sensor.
 6. The method of claim 1,comprising applying part of the supplemental data provided by thesupplemental sensor to a patch of the tracked patches.
 7. The method ofclaim 1, wherein the supplemental data is provided by the supplementalsensor in the form of a point cloud.
 8. The method of claim 4, whereinthe part of the supplemental data is a point of a point cloud.
 9. Themethod of claim 1, wherein the supplemental sensor is a radar sensor.10. The method of claim 4, wherein the equivalent scale ratio isdetermined from distance information and radial velocity information ofa point of the radar point cloud.
 11. The method of claim 1, comprising,for a point that is considered being located close to a patch of thetracked patches, comparing an equivalent scale ratio of the point with ascale ratio of the patch obtained from patch tracking to determine, ifthe scale ratios match.
 12. The method of claim 1, comprising discardingthose points of points that are considered being located close to apatch of the tracked patches, whose equivalent scale ratios do not matchwith a scale ratio of the patch obtained from affine patch tracking. 13.The method of claim 1, comprising performing object segmentation on theimage data captured by the camera.
 14. The method of claim 1, comprisingaveraging supplemental data related to patches associated with anobject, and associating the averaged information with the object. 15.The method of claim 1, wherein the supplemental data comprises Time toCollision information.
 16. The method of claim 1, wherein the sensorfusion is applied in an automotive context.
 17. A device comprisingcircuitry, the circuitry being configured to execute instructions, theinstructions, when executed on the circuitry, performing patch trackingon image data provided by a camera to determine tracked patches, and afusion of the image data obtained from the camera with supplemental dataprovided by a supplemental sensor based on the tracked patches.
 18. Asystem comprising a camera, a supplemental sensor, and the device ofclaim 17, the device being configured to perform patch tracking on imagedata provided by the camera to determine tracked patches, and to performa fusion of the image data obtained from the camera with supplementaldata provided by the supplemental sensor based on the tracked patches.